1. Field of the Invention
The present invention relates generally to systems and methods useful in connection with the quick replay of data captured during events, and more particularly to the quick video replay of events such as may occur during a professional sporting contest, such as professional football or the like, including for example the playback of video of an event in a reverse mode.
2. Background of the Invention
Just about everyone who has watched professional sports in recent years has seen a game stopped for a “video replay,” such as a stoppage of game play when one or more of the game officials will watch one or more monitors to see a replay of a play or event that recently occurred. Such replays may be to check on official rulings as to whether a player was in bounds or out of bounds when something happened (e.g., a catch, a fumble, a tackle, etc.), whether a player had possession of a ball at a certain moment in time or location on a playing field, whether a play was completed in time, and so forth. Typically, such replays have been of video or audiovisual footage taken during game play.
Conventional video systems allow for digitally recording and storing video images of a game during play. For example, there are a number of digital video cameras that are conventional, there are a number of digital video and audiovisual file formats in conventional use, and so forth. In conventional video storage and retrieval systems, it is typical to write the digital video files to a hard drive or disk. In such systems, when a video replay is desired, it is conventional to retrieve the desired video footage by reading it from the hard drive and then playing the same. In conventional systems, this often adds at least several seconds of delay to the replay; often as much as nine or ten additional seconds or possibly even longer. While this may not seem like a great delay, this additional nine or ten seconds is in addition to conventional delays caused by system constraints for recording, transmitting, and broadcasting the video signals, which themselves often result in a nine or ten second delay from when the event actually happens to when it can be played in replay. These delays may vary somewhat depending on the equipment and transmission systems involved and the bit rates involved, and may be shorter or longer in some systems. Moreover, conventional systems and methods for replaying video quickly in reverse-action mode by reading the video file from hard drive often do not provide a smooth and clear video image, but instead often provide choppy and sometimes poor image quality.
With conventional computer systems for video capture and playback, as noted above, there is often a significant delay between when the recorded event occurs and when it can be played back in replay. On a computer system, after the video is read from an external source it is typically first written to disk and then read back for playback.
In most conventional computer systems, the operating system (OS) utilizes buffers for temporary storage and assistance with file input and output (IO) operations in order to increase performance. In the case of storage on disks, a conventional operating system will usually, rather than writing data immediately to the disk, write the data to a buffer first and then later push the data to the disk when the disk is ready for writing. This approach typically enables better system performance. The software application making the write operation will not block access while waiting for disk to complete the write with this approach. Thus, this allows the system to continue processing data and also to write the video data to the buffer without blocking or requiring the write operation to first flush the data to disk. The use of conventional write buffers allow the operating system to continue processing data and write the data when the disk is ready.
Disabling the use of a write buffer usually leads to application performance degradation. When data is written directly to the disk without the data being stored in an intermediary write buffer, the software application usually blocks access while waiting for the disk to complete the write operation. This delay is often so large that incoming video data often can be dropped and permanently lost, as the application does not have enough time to read the new data. Further, conventional disks may have their own on-board cache. So even though a disk may report a write as complete, the data may still be in the disk's cache and not actually stored on the disk in non-volatile form.
One consequence of writing incoming data to a conventional write buffer is that data written into the write buffer is not immediately available for reading. Even though the operating system indicates the write operation has completed, the data will not be available for reading until the data has been flushed to disk. Although read operations in such situations do not necessarily fail, they nonetheless often return invalid or corrupt data if they reach the end of the disk file and try to read data still in the buffer. In the Windows operating system available from Microsoft Corporation, for example, the video data file's length will be updated even though the data file has not yet been flushed to disk. An attempt to read the latest data from disk in this situation results in a return of invalid data (e.g., all 0's). One approach to deal with this problem is to ensure that the read operation for the data to be played back does not read too closely to the end of the file. Otherwise, the system risks reading data that is present in the write buffer but not yet written to the disk. For highly compressed video data streams, keeping the read operation a few megabytes behind the end of the file will result in a delay of at least a few seconds behind the video data stream in conventional systems. It is believed that as video compression continues to improve, this latency will increase. For example, with conventional AVC/H.264 coding, the delay could get as long as 10 seconds, and perhaps longer depending on, for example, the bit rate of transmission.
Besides avoiding increased delay or latency, as well as corrupted or incomplete file storage, there are additional important considerations for live event analysis via video playback. For example, the ability to seek to certain points in time and to provide so-called “trick” play video playback with full reverse playback capabilities can be important. “Trick” play usually refers to the system's ability to play video at rates other than the normal 1× rate, including playback of the video in slow motion, fast forward, and reverse motion (slow and fast) modes.
In conventional video capture and storage systems and methods, key frames are often used. Key frames are fully encoded pictures. To achieve high compression ratios, inter-frame video codecs will often encode data for one full picture, a key frame, and store only the delta information for subsequent pictures. For example, if picture 1 is encoded as a key frame, picture 2 is stored as the differences between picture 1 and 2. Picture 3 is stored as the differences between picture 2 and 3, and so on. In systems using this approach, if one wishes to replay picture 3, the system must first decode pictures 1 and 2 before it can decode picture 3. Another alternative conventional approach involves intra-frame compression schemes in which every picture is stored as a full picture and is itself a key frame.
Key frames are sometimes referred to as I-frames. In addition to the key frames in a compressed video data file, a group of pictures (GOP) usually includes either or both of P-frames and B-frames. A P-frame (sometimes referred to as a predicted picture) is used to generate an image by using a prediction made from an earlier image, usually a key frame (or I-frame), although a P-frame can also be based on one or more earlier P-frames. Thus, a P-frame need only store or reflect the difference between the key-frame and the image to be generated from the P-frame.
A B-frame (sometimes referred to as a bidirectionally predicted picture or image) is a frame in which the image is predicted or interpolated from earlier and/or later frames. As noted, the I-, P-, and B-frames can be organized as a group of pictures (GOP). A GOP pattern can be considered long or short. Typically, a long-GOP pattern includes several P- and B-frames between each I-frame. A variety of video data compression formats have been published as standards, including the H.264 format, first published by International Telecommunication Union (ITU) in 2003 or so as ITU-T H.264, “Advanced video coding for generic audiovisual services,” which is hereby incorporated by reference as if fully set forth here.
In addition to inter-frame compression techniques for audiovisual data files, it is conventional to use intra-frame compression techniques as a possible alternative. Intra-frame compression techniques are used to create stand-alone frames (sometimes called I-frames). (P- and B-frames are not stand-alone frames.) Generally, intra-frame compression involves coding an image or frame with fewer bits than the original, such as for areas of similar color or texture.
In conventional video capture and storage systems a key image index was created and this index was shared with the reading process through disk. This approach resulted in the inability of the playback system to provide reduced time delays. In addition, the index had to always match the available data. To guard against video data still being in the write buffer, the index had to take a conservative approach and report a shorter file duration to prevent the application from reading too close to the end of the file. This resulted in an overall degradation in performance of the video playback, especially in situations requiring quick analysis of live events. A noticeable delay of even a few additional seconds during a televised broadcast of a game watched by millions of people is undesirable for a number of reasons. Moreover, for video data in formats using a long-GOP approach, providing a video playback in reverse mode quickly and with an acceptable quality is extremely difficult due to the encoding and compression algorithms and the resulting decoding needed for such video files.
A system for the replay of recordings of live events is described in U.S. Published Patent Application No. 2010/0195978 A1, published on Aug. 5, 2010. This application describes the use of instant replay in connection with the officiating in sports events, including NFL football games. This application also describes several shortcomings with conventional systems, including the labor intensive nature of the video replay process and the need for quickly providing such replays. Published U.S. Patent Application No. 2010/0195978 A1 is hereby incorporated by reference as if fully set forth herein.
A system using key frames to allow video replays of computer game action is described in U.S. Pat. No. 6,699,127 B1, issued on Mar. 2, 2004 to Lobb et al. This patent describes a system in which a video game user may choose to replay a portion of the game and in response to user's command the system “rewinds” the game by three seconds and the game replay begins from the key frame nearest the point three seconds earlier. The patent also describes the use of a wrapper buffer to store each key frame generated during the course of the game. U.S. Pat. No. 6,699,127 B1 is hereby incorporated by reference as if fully set forth herein.